When a software application runs on a computing device, a processor executes machine-level instructions into which high level programming code of the application has been translated (e.g., by a compiler and/or an interpreter, etc.). The pre-defined set of machine-level instructions that a particular processor can execute is the processor's instruction set. The processor typically fetches from memory, sequentially, the machine-level instructions corresponding to the functionality of a software application, and then executes the instructions sequentially.
However, the processor may encounter an instruction that transfers control of the program execution to another instruction or set of instructions stored in another memory location. For example, a call instruction causes the processor to transfer control of the program execution to a subroutine, such as a function or a procedure. In this case, the called subroutine will have among its instructions at least one return (“ret”) instruction. The return instruction causes the processor to return control back to the instruction that immediately follows the call instruction in the normal sequence of instructions. The memory location of the instruction to which control should return after a call instruction may be referred to as the return address. Some examples of control transfer instructions include call, jump (“jmp”), interrupt (“int”), and return from interrupt (“iret”).
The processor uses a call stack data structure to keep track of the program control flow when a call instruction is encountered. To do this, the call instruction pushes the return address onto the top of the stack. A stack pointer is used to keep track of the memory location of the top of the stack. The stack pointer is stored in a processor register. The return instruction uses the stack pointer to retrieve the return address from the top of the stack. The return instruction transfers control to the instruction found at the memory location of the return address.
Return-oriented programming (ROP) refers to a category of computer security exploits that allow an attacker to take control of the call stack and redirect the program flow to malicious code. A ROP attack may be enabled by a buffer overflow. A buffer overflow may occur, for example, as a result of a bug in an application program. When the buffer overflow occurs, the return address on the call stack may be overwritten with a different memory address. When this occurs, program flow can be diverted to an address at which an attacker's code (which may be referred to as a “gadget”) is located. Gadgets can be chained together to perform a series of functions that provide useful information to the attacker. Another example of a ROP attack is the stack pivoting exploit, which exposed vulnerabilities in some versions of the ADOBE ACROBAT and ADOBE READER software applications that allowed the stack pointer register to be manipulated by an adversary. This allowed the adversary to redirect the stack location in memory to a portion of the memory heap controlled by the adversary, effectively creating an alternative malicious stack. This attack was designed to trick users into clicking on a malicious PDF file delivered in an email message.